[On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints. #19770

AdamLouly · 2024-03-04T23:46:04Z

Description

Modifications to support 2GB+ checkpoint & Upgrading Flatbuffers

Motivation and Context

This PR includes changes that will make ort handle 2GB+ checkpoints.
To do that we need to upgrade flatbuffers to 23.5.9 - google/flatbuffers#7945

Modified the commitHash and the hash for the new version
Removed the patch for rust generator's unused variable warning as it is no longer producing this - Check it out here
Updated the VerifyField calls with alignment values that were introduced in the new version.

cmake/deps.txt

edgchen1 · 2024-03-06T03:49:47Z

A few places that may need updating:

onnxruntime/onnxruntime/core/flatbuffers/schema/README.md

Lines 23 to 24 in a788514

    
           It is possible to use another flatc as well, e.g., from a separate installation. Note that ONNX Runtime uses 
        
           FlatBuffers 1.12.

onnxruntime/cmake/external/onnxruntime_external_deps.cmake

Lines 101 to 108 in a788514

    
           #flatbuffers 1.11.0 does not have flatbuffers::IsOutRange, therefore we require 1.12.0+ 
        
           FetchContent_Declare( 
        
               flatbuffers 
        
               URL ${DEP_URL_flatbuffers} 
        
               URL_HASH SHA1=${DEP_SHA1_flatbuffers} 
        
               PATCH_COMMAND ${ONNXRUNTIME_FLATBUFFERS_PATCH_COMMAND} 
        
               FIND_PACKAGE_ARGS 1.12.0...<2.0.0 NAMES Flatbuffers 
        
           )

onnxruntime/js/web/package.json

Line 14 in a788514

"flatbuffers": "^1.12.0",

onnxruntime/js/web/package-lock.json

Line 12 in a788514

"flatbuffers": "^1.12.0",

@fs-eire would the onnxruntime-web flatbuffers dependency need to be updated as well?

sumitsays · 2024-03-08T20:11:14Z

From DirectML perspective, it looks good to me.

fs-eire · 2024-03-08T22:49:53Z

ONNX Runtime Web CI failed. This is because this PR does partially upgrade for flatbuffer for ort-web.

Please either revert this change for ort-web, or perform a full upgrade

Use npm install flatbuffers@^24.3.7 to install so that packages-lock.json will be refreshed with dependencies version
Follow instructions in js/web/onnxjs/ort-schema/flatbuffers/README.md to generate new flatbuffer files for v24.3.7 and update the files in js/web/onnxjs/ort-schema/flatbuffers/

We don't necessarily need this change for ort-web because only WebGL is using this flatbuffer library as dependencies, and we do not have plan to support on device training for WebGL backend. For Wasm/CPU/WebGPU, this library is not needed.

AdamLouly · 2024-03-08T23:01:48Z

ONNX Runtime Web CI failed. This is because this PR does partially upgrade for flatbuffer for ort-web.

Please either revert this change for ort-web, or perform a full upgrade

Use npm install flatbuffers@^24.3.7 to install so that packages-lock.json will be refreshed with dependencies version

Follow instructions in js/web/onnxjs/ort-schema/flatbuffers/README.md to generate new flatbuffer files for v24.3.7 and update the files in js/web/onnxjs/ort-schema/flatbuffers/

We don't necessarily need this change for ort-web because only WebGL is using this flatbuffer library as dependencies, and we do not have plan to support on device training for WebGL backend. For Wasm/CPU/WebGPU, this library is not needed.

Thanks for clarifying, in this case we can just revert ort-web upgrade.

tools/ci_build/github/azure-pipelines/templates/jobs/win-ci-vs-2022-job.yml

cmake/external/onnxruntime_external_deps.cmake

…new flatbuffer version

…m/microsoft/onnxruntime into adamlouly/update_ckpt_flatbuffers

snnn

LGTM. Thank you!

snnn

In file included from /mnt/vss/_work/1/s/include/onnxruntime/core/graph/graph.h:24, you need to add some guards to disable external warnings. Like what we do in https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/graph/onnx_protobuf.h . Otherwise the web pipelines won't pass

include/onnxruntime/core/graph/graph.h

pranavsharma

Reviewing for admin.

AdamLouly · 2024-03-14T23:36:21Z

Thanks for the review everyone!

upgrade flatbuffers

80a7386

edgchen1 reviewed Mar 5, 2024

View reviewed changes

cmake/deps.txt Outdated Show resolved Hide resolved

AdamLouly added 4 commits March 5, 2024 19:00

upgrade to recent version

b205310

fix patch

6e4f746

fix patch

7e34132

fix directml pipeline

3b4161e

update js and others

874dbc3

AdamLouly marked this pull request as ready for review March 6, 2024 19:47

AdamLouly requested review from a team as code owners March 6, 2024 19:47

AdamLouly requested a review from askhade March 6, 2024 19:48

AdamLouly added 2 commits March 6, 2024 20:26

remove items using powershell

e66fb65

resolve directML and update publish tests

a1b590d

AdamLouly requested a review from skottmckay March 8, 2024 03:08

revert ort web upgrade

2e4d62a

edgchen1 reviewed Mar 11, 2024

View reviewed changes

tools/ci_build/github/azure-pipelines/templates/jobs/win-ci-vs-2022-job.yml Outdated Show resolved Hide resolved

edgchen1 reviewed Mar 11, 2024

View reviewed changes

tools/ci_build/github/azure-pipelines/templates/jobs/win-ci-vs-2022-job.yml Outdated Show resolved Hide resolved

redefine publish tests

92f343e

edgchen1 reviewed Mar 12, 2024

View reviewed changes

cmake/external/onnxruntime_external_deps.cmake Outdated Show resolved Hide resolved

sumitsays and others added 4 commits March 12, 2024 13:26

Regenerate DmlGraph specific flatbuffer genereated C++ files for the …

a7b1d5e

…new flatbuffer version

Merge branch 'adamlouly/update_ckpt_flatbuffers' of https://github.co…

4d17a9a

…m/microsoft/onnxruntime into adamlouly/update_ckpt_flatbuffers

change version, remove publish tests

dc94b71

Merge branch 'adamlouly/update_ckpt_flatbuffers' of https://github.co…

596350e

…m/microsoft/onnxruntime into adamlouly/update_ckpt_flatbuffers

snnn previously approved these changes Mar 14, 2024

View reviewed changes

snnn reviewed Mar 14, 2024

View reviewed changes

ignore web warnings

8d4e000

AdamLouly dismissed snnn’s stale review via 8d4e000 March 14, 2024 00:24

edgchen1 reviewed Mar 14, 2024

View reviewed changes

include/onnxruntime/core/graph/graph.h Outdated Show resolved Hide resolved

AdamLouly added 2 commits March 14, 2024 10:06

add common flatbuffers and ignore

1b09a9f

add endif

1dc7431

edgchen1 approved these changes Mar 14, 2024

View reviewed changes

snnn approved these changes Mar 14, 2024

View reviewed changes

pranavsharma approved these changes Mar 14, 2024

View reviewed changes

AdamLouly merged commit 3255813 into main Mar 14, 2024
92 of 95 checks passed

AdamLouly deleted the adamlouly/update_ckpt_flatbuffers branch March 14, 2024 23:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints. #19770

[On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints. #19770

AdamLouly commented Mar 4, 2024

edgchen1 commented Mar 6, 2024

sumitsays commented Mar 8, 2024

fs-eire commented Mar 8, 2024

AdamLouly commented Mar 8, 2024

snnn left a comment

snnn left a comment

pranavsharma left a comment

AdamLouly commented Mar 14, 2024

[On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints. #19770

[On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints. #19770

Conversation

AdamLouly commented Mar 4, 2024

Description

Motivation and Context

edgchen1 commented Mar 6, 2024

sumitsays commented Mar 8, 2024

fs-eire commented Mar 8, 2024

AdamLouly commented Mar 8, 2024

snnn left a comment

Choose a reason for hiding this comment

snnn left a comment

Choose a reason for hiding this comment

pranavsharma left a comment

Choose a reason for hiding this comment

AdamLouly commented Mar 14, 2024